{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 4.5 实践：基于前馈神经网络完成鸢尾花分类\n",
    "\n",
    "在本实践中，我们继续使用第三章中的鸢尾花分类任务，将Softmax分类器替换为本章介绍的前馈神经网络。\n",
    "在本实验中，我们使用的损失函数为交叉熵损失；优化器为随机梯度下降法；评价指标为准确率。\n",
    "\n",
    "### 4.5.1 小批量梯度下降法\n",
    "在梯度下降法中，目标函数是整个训练集上的风险函数，这种方式称为**批量梯度下降法（Batch Gradient Descent，BGD）**。 批量梯度下降法在每次迭代时需要计算每个样本上损失函数的梯度并求和。当训练集中的样本数量$N$很大时，空间复杂度比较高，每次迭代的计算开销也很大。\n",
    "\n",
    "为了减少每次迭代的计算复杂度，我们可以在每次迭代时只采集一小部分样本，计算在这组样本上损失函数的梯度并更新参数，这种优化方式称为\n",
    "小批量梯度下降法（Mini-Batch Gradient Descent，Mini-Batch GD）。\n",
    "\n",
    "第$t$次迭代时，随机选取一个包含$K$个样本的子集$\\mathcal{B}_t$，计算这个子集上每个样本损失函数的梯度并进行平均，然后再进行参数更新。\n",
    "$$\n",
    "\\theta_{t+1} \\leftarrow \\theta_t  - \\alpha \\frac{1}{K} \\sum_{(\\boldsymbol{x},y)\\in \\mathcal{S}_t} \\frac{\\partial \\mathcal{L}\\Big(y,f(\\boldsymbol{x};\\theta)\\Big)}{\\partial \\theta},\n",
    "$$\n",
    "其中$K$为**批量大小(Batch Size)**。$K$通常不会设置很大，一般在$1\\sim100$之间。在实际应用中为了提高计算效率，通常设置为2的幂$2^n$。\n",
    "\n",
    "在实际应用中，小批量随机梯度下降法有收敛快、计算开销小的优点，因此逐渐成为大规模的机器学习中的主要优化算法。\n",
    "此外，随机梯度下降相当于在批量梯度下降的梯度上引入了随机噪声。在非凸优化问题中，随机梯度下降更容易逃离局部最优点。\n",
    "\n",
    "小批量随机梯度下降法的训练过程如下：\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/49287f7b41234d718fcc79b3f8edc64f366b05f460fb49cea88805777d508d82)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.5.1.1 数据分组\n",
    "为了小批量梯度下降法，我们需要对数据进行随机分组。目前，机器学习中通常做法是构建一个**数据迭代器**，每个迭代过程中从全部数据集中获取一批指定数量的数据。\n",
    "\n",
    "数据迭代器的实现原理如下图所示：\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/337735d2bf6c49dda5415d791f6a320ffe739a809b3541faa26bfb8331506cb3)\n",
    "\n",
    "\n",
    "1. 首先，将数据集封装为Dataset类，传入一组索引值，根据索引从数据集合中获取数据；\n",
    "1. 其次，构建DataLoader类，需要指定数据批量的大小和是否需要对数据进行乱序，通过该类即可批量获取数据。\n",
    "\n",
    "在实践过程中，通常使用进行参数优化。在飞桨中，使用`paddle.io.DataLoader`加载minibatch的数据，\n",
    "`paddle.io.DataLoader`  API可以生成一个迭代器，其中通过设置`batch_size`参数来指定minibatch的长度，通过设置shuffle参数为True，可以在生成`minibatch`的索引列表时将索引顺序打乱。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.5.2 数据处理\n",
    "\n",
    "构造IrisDataset类进行数据读取，继承自`paddle.io.Dataset`类。`paddle.io.Dataset`是用来封装 Dataset的方法和行为的抽象类，通过一个索引获取指定的样本，同时对该样本进行数据处理。当继承`paddle.io.Dataset`来定义数据读取类时，实现如下方法：\n",
    "\n",
    "* `__getitem__`：根据给定索引获取数据集中指定样本，并对样本进行数据处理；\n",
    "* `__len__`：返回数据集样本个数。\n",
    "\n",
    "代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import paddle\n",
    "import paddle.io as io\n",
    "from nndl.dataset import load_data\n",
    "\n",
    "class IrisDataset(io.Dataset):\n",
    "    def __init__(self, mode='train', num_train=120, num_dev=15):\n",
    "        super(IrisDataset, self).__init__()\n",
    "        # 调用第三章中的数据读取函数，其中不需要将标签转成one-hot类型\n",
    "        X, y = load_data(shuffle=True)\n",
    "        if mode == 'train':\n",
    "            self.X, self.y = X[:num_train], y[:num_train]\n",
    "        elif mode == 'dev':\n",
    "            self.X, self.y = X[num_train:num_train + num_dev], y[num_train:num_train + num_dev]\n",
    "        else:\n",
    "            self.X, self.y = X[num_train + num_dev:], y[num_train + num_dev:]\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        return self.X[idx], self.y[idx]\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "paddle.seed(12)\n",
    "train_dataset = IrisDataset(mode='train')\n",
    "dev_dataset = IrisDataset(mode='dev')\n",
    "test_dataset = IrisDataset(mode='test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "length of train set:  120\n"
     ]
    }
   ],
   "source": [
    "# 打印训练集长度\n",
    "print (\"length of train set: \", len(train_dataset))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "#### 4.5.2.2 用DataLoader进行封装"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 批量大小\n",
    "batch_size = 16\n",
    "\n",
    "# 加载数据\n",
    "train_loader = io.DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
    "dev_loader = io.DataLoader(dev_dataset, batch_size=batch_size)\n",
    "test_loader = io.DataLoader(test_dataset, batch_size=batch_size)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.5.3 模型构建\n",
    "\n",
    "构建一个简单的前馈神经网络进行鸢尾花分类实验。其中输入层神经元个数为4，输出层神经元个数为3，隐含层神经元个数为6。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from paddle import nn\n",
    "\n",
    "# 定义前馈神经网络\n",
    "class Model_MLP_L2_V3(nn.Layer):\n",
    "    def __init__(self, input_size, output_size, hidden_size):\n",
    "        super(Model_MLP_L2_V3, self).__init__()\n",
    "        # 构建第一个全连接层\n",
    "        self.fc1 = nn.Linear(\n",
    "            input_size,\n",
    "            hidden_size,\n",
    "            weight_attr=paddle.ParamAttr(initializer=nn.initializer.Normal(mean=0.0, std=0.01)),\n",
    "            bias_attr=paddle.ParamAttr(initializer=nn.initializer.Constant(value=1.0))\n",
    "        )\n",
    "        # 构建第二全连接层\n",
    "        self.fc2 = nn.Linear(\n",
    "            hidden_size,\n",
    "            output_size,\n",
    "            weight_attr=paddle.ParamAttr(initializer=nn.initializer.Normal(mean=0.0, std=0.01)),\n",
    "            bias_attr=paddle.ParamAttr(initializer=nn.initializer.Constant(value=1.0))\n",
    "        )\n",
    "        # 定义网络使用的激活函数\n",
    "        self.act = nn.Sigmoid()\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        outputs = self.fc1(inputs)\n",
    "        outputs = self.act(outputs)\n",
    "        outputs = self.fc2(outputs)\n",
    "        return outputs\n",
    "\n",
    "fnn_model = Model_MLP_L2_V3(input_size=4, output_size=3, hidden_size=6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.5.4 完善Runner类\n",
    "\n",
    "基于RunnerV2类进行完善实现了RunnerV3类。其中训练过程使用自动梯度计算，使用`DataLoader`加载批量数据，使用随机梯度下降法进行参数优化；模型保存时，使用`state_dict`方法获取模型参数；模型加载时，使用`set_state_dict`方法加载模型参数.\n",
    "\n",
    "由于这里使用随机梯度下降法对参数优化，所以数据以批次的形式输入到模型中进行训练，那么评价指标计算也是分别在每个批次进行的，要想获得每个epoch整体的评价结果，需要对历史评价结果进行累积。这里定义`Accuracy`类实现该功能。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from paddle.metric import Metric\n",
    "\n",
    "class Accuracy(Metric):\n",
    "    def __init__(self, is_logist=True):\n",
    "        \"\"\"\n",
    "        输入：\n",
    "           - is_logist: outputs是logist还是激活后的值\n",
    "        \"\"\"\n",
    "\n",
    "        # 用于统计正确的样本个数\n",
    "        self.num_correct = 0\n",
    "        # 用于统计样本的总数\n",
    "        self.num_count = 0\n",
    "\n",
    "        self.is_logist = is_logist\n",
    "\n",
    "    def update(self, outputs, labels):\n",
    "        \"\"\"\n",
    "        输入：\n",
    "           - outputs: 预测值, shape=[N,class_num]\n",
    "           - labels: 标签值, shape=[N,1]\n",
    "        \"\"\"\n",
    "\n",
    "        # 判断是二分类任务还是多分类任务，shape[1]=1时为二分类任务，shape[1]>1时为多分类任务\n",
    "        if outputs.shape[1] == 1: # 二分类\n",
    "            outputs = paddle.squeeze(outputs, axis=-1)\n",
    "            if self.is_logist:\n",
    "                # logist判断是否大于0\n",
    "                preds = paddle.cast((outputs>=0), dtype='float32')\n",
    "            else:\n",
    "                # 如果不是logist，判断每个概率值是否大于0.5，当大于0.5时，类别为1，否则类别为0\n",
    "                preds = paddle.cast((outputs>=0.5), dtype='float32')\n",
    "        else:\n",
    "            # 多分类时，使用'paddle.argmax'计算最大元素索引作为类别\n",
    "            preds = paddle.argmax(outputs, axis=1, dtype='int64')\n",
    "\n",
    "        # 获取本批数据中预测正确的样本个数\n",
    "        labels = paddle.squeeze(labels, axis=-1)\n",
    "        batch_correct = paddle.sum(paddle.cast(preds==labels, dtype=\"float32\")).numpy()[0]\n",
    "        batch_count = len(labels)\n",
    "\n",
    "        # 更新num_correct 和 num_count\n",
    "        self.num_correct += batch_correct\n",
    "        self.num_count += batch_count\n",
    "\n",
    "    def accumulate(self):\n",
    "        # 使用累计的数据，计算总的指标\n",
    "        if self.num_count == 0:\n",
    "            return 0\n",
    "        return self.num_correct / self.num_count\n",
    "\n",
    "    def reset(self):\n",
    "        # 重置正确的数目和总数\n",
    "        self.num_correct = 0\n",
    "        self.num_count = 0\n",
    "\n",
    "    def name(self):\n",
    "        return \"Accuracy\"\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "RunnerV3类的代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import paddle.nn.functional as F\n",
    "\n",
    "class RunnerV3(object):\n",
    "    def __init__(self, model, optimizer, loss_fn, metric, **kwargs):\n",
    "        self.model = model\n",
    "        self.optimizer = optimizer\n",
    "        self.loss_fn = loss_fn\n",
    "        self.metric = metric # 只用于计算评价指标\n",
    "\n",
    "        # 记录训练过程中的评价指标变化情况\n",
    "        self.dev_scores = []\n",
    "\n",
    "        # 记录训练过程中的损失函数变化情况\n",
    "        self.train_epoch_losses = [] # 一个epoch记录一次loss\n",
    "        self.train_step_losses = []  # 一个step记录一次loss\n",
    "        self.dev_losses = []\n",
    "        \n",
    "        # 记录全局最优指标\n",
    "        self.best_score = 0\n",
    "\n",
    "    def train(self, train_loader, dev_loader=None, **kwargs):\n",
    "        # 将模型切换为训练模式\n",
    "        self.model.train()\n",
    "\n",
    "        # 传入训练轮数，如果没有传入值则默认为0\n",
    "        num_epochs = kwargs.get(\"num_epochs\", 0)\n",
    "        # 传入log打印频率，如果没有传入值则默认为100\n",
    "        log_steps = kwargs.get(\"log_steps\", 100)\n",
    "        # 评价频率\n",
    "        eval_steps = kwargs.get(\"eval_steps\", 0)\n",
    "\n",
    "        # 传入模型保存路径，如果没有传入值则默认为\"best_model.pdparams\"\n",
    "        save_path = kwargs.get(\"save_path\", \"best_model.pdparams\")\n",
    "\n",
    "        custom_print_log = kwargs.get(\"custom_print_log\", None) \n",
    "       \n",
    "        # 训练总的步数\n",
    "        num_training_steps = num_epochs * len(train_loader)\n",
    "\n",
    "        if eval_steps:\n",
    "            if self.metric is None:\n",
    "                raise RuntimeError('Error: Metric can not be None!')\n",
    "            if dev_loader is None:\n",
    "                raise RuntimeError('Error: dev_loader can not be None!')\n",
    "            \n",
    "        # 运行的step数目\n",
    "        global_step = 0\n",
    "\n",
    "        # 进行num_epochs轮训练\n",
    "        for epoch in range(num_epochs):\n",
    "            # 用于统计训练集的损失\n",
    "            total_loss = 0\n",
    "            for step, data in enumerate(train_loader):\n",
    "                X, y = data\n",
    "                # 获取模型预测\n",
    "                logits = self.model(X)\n",
    "                loss = self.loss_fn(logits, y) # 默认求mean\n",
    "                total_loss += loss \n",
    "\n",
    "                # 训练过程中，每个step的loss进行保存\n",
    "                self.train_step_losses.append((global_step,loss.item()))\n",
    "\n",
    "                if log_steps and global_step%log_steps==0:\n",
    "                    print(f\"[Train] epoch: {epoch}/{num_epochs}, step: {global_step}/{num_training_steps}, loss: {loss.item():.5f}\")\n",
    "                \n",
    "                # 梯度反向传播，计算每个参数的梯度值\n",
    "                loss.backward() \n",
    "\n",
    "                if custom_print_log:\n",
    "                   custom_print_log(self)\n",
    "                \n",
    "                # 小批量梯度下降进行参数更新\n",
    "                self.optimizer.step()\n",
    "                # 梯度归零\n",
    "                self.optimizer.clear_grad()\n",
    "\n",
    "                # 判断是否需要评价\n",
    "                if eval_steps>0 and global_step>0 and \\\n",
    "                    (global_step%eval_steps == 0 or global_step==(num_training_steps-1)):\n",
    "\n",
    "                    dev_score, dev_loss = self.evaluate(dev_loader, global_step=global_step)\n",
    "                    print(f\"[Evaluate]  dev score: {dev_score:.5f}, dev loss: {dev_loss:.5f}\") \n",
    "\n",
    "                    # 将模型切换为训练模式\n",
    "                    self.model.train()\n",
    "\n",
    "                    # 如果当前指标为最优指标，保存该模型\n",
    "                    if dev_score > self.best_score:\n",
    "                        self.save_model(save_path)\n",
    "                        print(f\"[Evaluate] best accuracy performence has been updated: {self.best_score:.5f} --> {dev_score:.5f}\")\n",
    "                        self.best_score = dev_score\n",
    "\n",
    "                global_step += 1\n",
    "            \n",
    "            # 当前epoch 训练loss累计值 \n",
    "            trn_loss = (total_loss / len(train_loader)).item()\n",
    "            # epoch粒度的训练loss保存\n",
    "            self.train_epoch_losses.append(trn_loss)\n",
    "            \n",
    "        print(\"[Train] Training done!\")\n",
    "\n",
    "    # 模型评估阶段，使用'paddle.no_grad()'控制不计算和存储梯度\n",
    "    @paddle.no_grad()\n",
    "    def evaluate(self, dev_loader, **kwargs):\n",
    "        assert self.metric is not None\n",
    "\n",
    "        # 将模型设置为评估模式\n",
    "        self.model.eval()\n",
    "\n",
    "        global_step = kwargs.get(\"global_step\", -1) \n",
    "\n",
    "        # 用于统计训练集的损失\n",
    "        total_loss = 0\n",
    "\n",
    "        # 重置评价\n",
    "        self.metric.reset() \n",
    "        \n",
    "        # 遍历验证集每个批次    \n",
    "        for batch_id, data in enumerate(dev_loader):\n",
    "            X, y = data\n",
    "    \n",
    "            # 计算模型输出\n",
    "            logits = self.model(X)\n",
    "            \n",
    "            # 计算损失函数\n",
    "            loss = self.loss_fn(logits, y).item()\n",
    "            # 累积损失\n",
    "            total_loss += loss \n",
    "\n",
    "            # 累积评价\n",
    "            self.metric.update(logits, y)\n",
    "\n",
    "        dev_loss = (total_loss/len(dev_loader))\n",
    "        dev_score = self.metric.accumulate() \n",
    "\n",
    "        # 记录验证集loss\n",
    "        if global_step!=-1:\n",
    "            self.dev_losses.append((global_step, dev_loss))\n",
    "            self.dev_scores.append(dev_score)\n",
    "        \n",
    "        return dev_score, dev_loss\n",
    "    \n",
    "    # 模型评估阶段，使用'paddle.no_grad()'控制不计算和存储梯度\n",
    "    @paddle.no_grad()\n",
    "    def predict(self, x, **kwargs):\n",
    "        # 将模型设置为评估模式\n",
    "        self.model.eval()\n",
    "        # 运行模型前向计算，得到预测值\n",
    "        logits = self.model(x)\n",
    "        return logits\n",
    "\n",
    "    def save_model(self, save_path):\n",
    "        paddle.save(self.model.state_dict(), save_path)\n",
    "\n",
    "    def load_model(self, model_path):\n",
    "        model_state_dict = paddle.load(model_path)\n",
    "        self.model.set_state_dict(model_state_dict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.5.5 模型训练\n",
    "\n",
    "实例化RunnerV3类，并传入训练配置，代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import paddle.optimizer as opt\n",
    "\n",
    "lr = 0.2\n",
    "\n",
    "# 定义网络\n",
    "model = fnn_model\n",
    "\n",
    "# 定义优化器\n",
    "optimizer = opt.SGD(learning_rate=lr, parameters=model.parameters())\n",
    "\n",
    "# 定义损失函数。softmax+交叉熵\n",
    "loss_fn = F.cross_entropy\n",
    "\n",
    "# 定义评价指标\n",
    "metric = Accuracy(is_logist=True)\n",
    "\n",
    "runner = RunnerV3(model, optimizer, loss_fn, metric)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "使用训练集和验证集进行模型训练，共训练150个epoch。在实验中，保存准确率最高的模型作为最佳模型。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Train] epoch: 0/150, step: 0/1200, loss: 1.09929\n",
      "[Evaluate]  dev score: 0.40000, dev loss: 1.10371\n",
      "[Evaluate] best accuracy performence has been updated: 0.00000 --> 0.40000\n",
      "[Train] epoch: 12/150, step: 100/1200, loss: 1.18915\n",
      "[Evaluate]  dev score: 0.40000, dev loss: 1.08898\n",
      "[Evaluate]  dev score: 0.40000, dev loss: 1.09164\n",
      "[Train] epoch: 25/150, step: 200/1200, loss: 1.10245\n",
      "[Evaluate]  dev score: 0.33333, dev loss: 1.08986\n",
      "[Evaluate]  dev score: 0.40000, dev loss: 1.08724\n",
      "[Train] epoch: 37/150, step: 300/1200, loss: 1.09221\n",
      "[Evaluate]  dev score: 0.40000, dev loss: 1.07255\n",
      "[Evaluate]  dev score: 0.66667, dev loss: 1.03728\n",
      "[Evaluate] best accuracy performence has been updated: 0.40000 --> 0.66667\n",
      "[Train] epoch: 50/150, step: 400/1200, loss: 1.00845\n",
      "[Evaluate]  dev score: 0.73333, dev loss: 0.92129\n",
      "[Evaluate] best accuracy performence has been updated: 0.66667 --> 0.73333\n",
      "[Evaluate]  dev score: 0.93333, dev loss: 0.77246\n",
      "[Evaluate] best accuracy performence has been updated: 0.73333 --> 0.93333\n",
      "[Train] epoch: 62/150, step: 500/1200, loss: 0.60928\n",
      "[Evaluate]  dev score: 0.80000, dev loss: 0.63509\n",
      "[Evaluate]  dev score: 0.80000, dev loss: 0.54118\n",
      "[Train] epoch: 75/150, step: 600/1200, loss: 0.46621\n",
      "[Evaluate]  dev score: 0.80000, dev loss: 0.48350\n",
      "[Evaluate]  dev score: 1.00000, dev loss: 0.43852\n",
      "[Evaluate] best accuracy performence has been updated: 0.93333 --> 1.00000\n",
      "[Train] epoch: 87/150, step: 700/1200, loss: 0.33996\n",
      "[Evaluate]  dev score: 1.00000, dev loss: 0.41020\n",
      "[Evaluate]  dev score: 1.00000, dev loss: 0.38648\n",
      "[Train] epoch: 100/150, step: 800/1200, loss: 0.31987\n",
      "[Evaluate]  dev score: 1.00000, dev loss: 0.36471\n",
      "[Evaluate]  dev score: 0.93333, dev loss: 0.34849\n",
      "[Train] epoch: 112/150, step: 900/1200, loss: 0.36447\n",
      "[Evaluate]  dev score: 0.93333, dev loss: 0.31938\n",
      "[Evaluate]  dev score: 1.00000, dev loss: 0.30559\n",
      "[Train] epoch: 125/150, step: 1000/1200, loss: 0.31020\n",
      "[Evaluate]  dev score: 0.93333, dev loss: 0.28503\n",
      "[Evaluate]  dev score: 1.00000, dev loss: 0.27043\n",
      "[Train] epoch: 137/150, step: 1100/1200, loss: 0.23952\n",
      "[Evaluate]  dev score: 0.93333, dev loss: 0.25519\n",
      "[Evaluate]  dev score: 0.93333, dev loss: 0.24227\n",
      "[Evaluate]  dev score: 1.00000, dev loss: 0.23113\n",
      "[Train] Training done!\n"
     ]
    }
   ],
   "source": [
    "# 启动训练\n",
    "log_steps = 100\n",
    "eval_steps = 50\n",
    "runner.train(train_loader, dev_loader, \n",
    "            num_epochs=150, log_steps=log_steps, eval_steps = eval_steps,\n",
    "            save_path=\"best_model.pdparams\") "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "可视化观察训练集损失和训练集loss变化情况。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2349: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n",
      "  if isinstance(obj, collections.Iterator):\n",
      "/opt/conda/envs/python35-paddle120-env/lib/python3.7/site-packages/matplotlib/cbook/__init__.py:2366: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working\n",
      "  return list(data) if isinstance(data, collections.MappingView) else data\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAENCAYAAAAVPvJNAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzsvXmcHGWd+P/+VFVfc2euzGRyJ5ODkBAghMgh0SUCrqCA8IsblbAgCwrrgf5AVlx3VVRWBeRQ8WJxVRDxAERABYNCgAQCCUnIfU6uue8+qur5/lE9nbkyZ09PT8/zfr36le6qp5/+TPr41OcWpRQajUaj0XTHGG0BNBqNRpOeaAWh0Wg0ml7RCkKj0Wg0vaIVhEaj0Wh6RSsIjUaj0fSKVhAajUaj6RWtIDQajUbTK1pBaDQajaZXtILQaDQaTa9Yoy3AcCguLlbTp08fbTE0Gcrrr79eo5QqGY3X1p9tzUgy0M/2mFYQ06dPZ/369aMthiZDEZF9o/Xa+rOtGUkG+tnWLiaNRqPR9IpWEBqNRqPplZQoCBH5qYgcE5G3T3B+lYhsFJFNIvKyiJySCrk0Go1Gc2JSZUE8BFzYx/k9wHlKqYXAV4EHUyGURqPRaE5MSoLUSqkXRWR6H+df7vTwFWDySMuk0Wg0mr5JxxjENcCfRlsIjWYgDMB9KiLyPRHZGXejnpZqGTWaoZJWaa4i8h48BXFOH2uuA64DmDp16ojK47ouNTU1NDQ04DjOiL6WJrWYpklBQQHFxcUYxrCukx4C7gMePsH5i4DK+O1M4PvxfzWatCdtFISILAJ+DFyklKo90Tql1IPEYxRLlixJyrzUtv97i+A/z8GYEOpy/ODBg4gI06dPx+fzISLJeDnNKKOUIhaLcfToUQ4ePDisC43+3KfAB4GHlTfb9xURKRCRcqXU4SG/qOaE2EcacaqbCSz0vNTRnUdxm9q7rDGCPvwnVXjntx/BbQl3PZ8VwD+vPDUCd8NpaMNtCeObXDgqr9+dtHAxichU4LfAx5RS21P52vauOuo/9jtaf/JGj3Otra1UVFTg9/u1csggRAS/309FRQWtra0j/XIVwIFOjw/Gj/Um13Uisl5E1ldXV4+0XBlJ+LXdRNbvSTy2d1cTfetA19u2I4nzsZ1He5yP7TyaNHmU4w5qvVPdRPjFbahYengsUmJBiMivgOVAsYgcBP4T8AEopX4AfBkoAh6I/xDbSqklqZDN3lbj/bu59y/kMN0PmjQm3d7bkbCOxxuqLYJvTlnicdb7Tu5zffb7e8+od5vaCb+2m8DiaZjFOUOSxQ1HaXn0NaypRQROmYpZmN3vc4zsACpiYx9qwDetaEivm0xSlcX0kX7OXwtcmwpZumPvqAMgtkVfsWlGhCpgSqfHk+PHNElGuQrVHkOy/MPfzG9hH6zDKMwesoKwD9SDq7D31mCE/JjLZvW9/lgTbmsETAP7QO2QFUR01zFi248Qes98jKBvSHt0kF6XUKOAvd0Ld9hbqvHcxBpNUnkC+Hg8m2kZ0KjjDyODao8CYISGryCMoA+zNA/7QN2Q97AP1CJZfqwphcQO1Pb7+xJ75zCRdXuwphRiH6gb8u+Rva8Gt6kdCQz/+l8riB2eglAtUZwDjaMszdjgnXfeQUSG3UzumWeeQUSoqalJkmSpJ+4+XQvMFZGDInKNiFwvItfHlzwN7AZ2Aj8CPjlKomY8KuYguUEkJ5CU/aypRbh1rT2C2AOSxXaxq+qxphZhTS1CtURw69tOvN5V2AfrsCYX4ptWhArHcKqbB/+6jotd1YA1pTApcdO0yWJKFW5bFByFket9iOzttZhT83H2N2JvqcaaWjDKEg6f/j4Y06ZNY+/evUPev7KyksOHD1NcXDzkPTKFAbhPFfCpFIkzrjELssj98BlJ28+aUkRk3R7s/XX4T5o0qOfaRxrAdvFNKcQo8lxU9v7aE8YhnGNNqIjtKZTyAvwLKobkHnKONILtYE1JThbUuLMgGq7+A7X//AsAT0vvbyT4wbkAxE4QqB5rHD58OHF7/PHHAXjjjTcSx9atW9fr86LR6ID2N02TsrIyLGvcXV9oxhFmfghrevGQXDVmfojA4qmYZQVe/KEkF6fmxBaBvb8WDMGqKEACFsGlMzHyQidcf8J9DtSBaWCVJ+dCd9wpiOjaA0RfOoDbGMbeVQ8K/MsmY5RmY2dIoLqsrCxxKyz0riRKSkoSx0pKShLr/uu//ovrrruOwsJCVqxYAcC3v/1tFi1aRHZ2NpMmTeKjH/0ox44dS+zf3cXU8fi3v/0tF110EVlZWcyePZtf/vKXg5b9H//4B+eccw7BYJDCwkI+/vGPU1t7vCxm3759fOhDH6KoqIhQKMTs2bO55557Eud/85vfcMopp5CVlcWECRN417vexdtv91rkrMkwolsP0frc20mNJWa9Zz6+WaWDfp6RGyJw6jTE8n5iQ+cvIPRPJ51wvdvUjllegPg8ZaRchX24wfN4DOZ1i3LwL6hALHPQMvfGuLoEdJvCOAeaAIi+fAAVsQGw5hRhLSgZUCZTw2f+ROzNI/2uSya+xWUU3H3RiOz9ne98h1tvvZVXX301US0uItx9993MmDGDQ4cO8dnPfpaPfexjPPvss33udcstt/Ctb32Le++9lx/84AesXr2as846i4FORjtw4AAXXHABV1xxBT/84Q+pqanh+uuvZ+XKlfz5z38G4BOf+ASmafL888+Tn5/Prl27Egpk//79rFy5ku985ztccskltLe38/rrr2OayfmyaNIbp7YFt7416TVLynZQURsja2CxDbepHaexDWvSBMT0FER/7qKs8xeg7OO1D6o1TNszmwgsnUlgQa9lM73ir5w44LUDYVwpiM4WQuTFfRiFnglnVRbhO6mEtp9vRCk1rorizj33XG677bYux26++ebE/RkzZnDPPfdw1llnUVtbS1HRiVPvPvvZz3LZZZcBcMcdd3DvvfeyZs2aASuI733ve0ycOJEf//jHCffVQw89xLJly3jttddYunQp+/bt4+qrr+aUU7z89c57V1VV4bouV155JeXlXiXsSSed+KpNk1m4bdHkpLh2QilFy29fx5qYR+i8eQN6TnTXMaJv7Sdn5bKEggAIv7EX1RoldO6cXp/X+arfyA1hFGRh768dsIJwGtowQj4kMLzU1s6MKwXREWMwJuUSXbMXa0EpRmk2Rn4Q66QSVFMEt6oJoywH51Dv/sKRupIfLZYuXdrj2F/+8he+9a1v8c4779DQ0IDretWg+/bt61NBLF68OHHf7/dTXFzM0aMDr0rdvHkzZ511VpfYxtKlSwkGg2zevJmlS5fyuc99jhtvvJE//OEPLF++nA984AOcffbZAJxxxhmcd955zJ07lxUrVrB8+XIuu+wyKioGfgWmGbuotihGkjKYOhARrPICYvtrCbouMoDiSnt/LWZpXk+rIeYQ23OM4LJZiO+4Mmj789sYhTkET5/eZbk1tYjopgOoSGxAP/rhl3agXJeci08d0N82EMZVDCK2+RgSssj66CKi6w4Re+sIVqXno/ct8PyMsU3HqLvsUY7Nvx/GQV1EdnbXrIqdO3fygQ98gLlz5/Loo4+yfv16HnvsMaD/ILbf3/XqTUQSyiVZ/Nu//Rt79uzhmmuuYf/+/axYsYJrr/VqLC3L4vnnn+e5557j1FNP5ZFHHqGysjLhntJkNqotknQLAsCaWghRG+doU79r3ZYwbl0r1pSeF1LWlCJwFPah+uPrw1Hsg/WI0dNrYU0tBAX2wfoe53q8bjiGU92EleQeTuNKQdhvH8OaX0Jg+XSwXWLrDmFVem+kdZIXuG244SnCT25HtcVQdnJ/3MYCr776KrFYjLvvvpuzzjqLuXPncuRIamIuCxYs4OWXX8a27cSx1157jXA4zMknH2+ZMHnyZK699lp+8Ytf8MADD/DTn/6USCQCeEpp2bJlfOlLX+Kll15i6dKlPPTQQymRXzN6KKUwJmQPqJ3FYLEmTQBTBlQ017HGmtrzh9osywO/2WUf+0B9fH1PhWIW5yIhXxeFcsLXPVgHiqQ3+RtXCiK2uRprQQn+s6dAXGNbc7w3xizJxijOwtnXiP/ceHfPcagg5syZg+u63HXXXezZs4fHH3+cb3zjGyl57U9/+tMcPXqUa6+9ls2bN7NmzRquvvpqzj//fM44w8tvv/7663nmmWfYtWsXb7/9Nr///e+ZNWsWgUCAv/3tb9xxxx289tpr7N+/n+eee44tW7boOMQ4QETIvmgR/nmDq1cY0N4+M+Fm6i9Dyj7aiJEfwszP6rmPYWBNjldJu94+9n6v2troRbGJCFkXLSJ4dmW/MtoH6pCQD2OIbUFOxLhREG5DO+6hZnwnl2LkBfGd6jX06nAxAQQvm09o1UIm/O+lACg7811M3TnjjDP47ne/yz333MNJJ53Evffey1133ZWS1548eTLPPvssO3bs4PTTT+fSSy9lyZIlPPLII4k1juNw0003cfLJJ3PeeefhOA5PPvkkABMmTODFF1/k4osvprKykuuuu45rrrmGW265JSXyazKXwKnTyHrP/H7Xhc6bR9aFC0943jezFGt6MdiOV219yKu2PlFijJmf1W/cQ7nxqu3Jyame7oyM5f5DS5YsUQNt9xB5aT815/yUoqf+heA/z6Hx5mdp+e5aSt+6Ht+isi5rle1wKPg1Gv5+CQveNbCAj4o5OFXNoBTm9IJxlQk1ltm6dSvz5/f+xReR11PVVbg7g/lsayB2oJbIuj1knb9gSAVmo4GKxIhsPIg1tRBrYv4J14Xf2ItYJoFFU064xmlsAwQzf2B/+0A/2+Mmi8ne7BV6mXM9iyHrX0/FqWnDml/SY61YJuaU/BO6mJTjgqvAUaiIjWqL4RxpAcdTtkZhCMkPjtBfotFouqOaw7iN7eAfuZoX+0gjztFGAqf0PmAq/NpuMITgkhl97qOUwq1rxSzKIXhG32sB3LpWnPpW/Asn92lpjAQZqSBiW6uJbTpK1pXHA5uxzdUY0/Nof2U7Tn0T/oWTKYy7knrDnJbfJUitog5uTRtufTuq3e6xXnL9mFPycXbV4VQ1IXmBxJuplEI1hMEQDK04NJqk47ZFwZCk1gB0xznSSGTDPnxzynp0jFWuIrrjKL5egtPdiW07THjtLoJnzcZXObFfF5I1tQj7QB1ufVuvQfjwq7sSPZySTUbGIJq+/AL1V/2e2Jbj7SHszccwpxfgm1pE9O0qWh5bT/j1vahIrNc9rOkFXSwIe0etVxthGpiTcjGn5mFOL8CaW4Rv0USsOUUYWT6MSbmoNhtV73WAVLaDs7see1c9zj7dLVajGQlUWxQJjezkx47MpN6ymZxjTRC1e01v7bFPhbdP+OWdxHb3372ho/Gevb/nJGansY3olkO4DSfuFDscMtKCKLjv/Rxbs5f6Vb+l5NVrwVXENh0j+P5KQu+ei3/RFCIb9hHdeIDYjiPkfPgMxDKJbDqI2xJGtUcxzyxCcn24rREkYKHabcypuUi2H0S8qxVDwDS6VEsahSHcIy04B5twattQLVFwFZLjR7VEvZbEPt36QTM+iWyuwsgJ4JuW3E7AbnsUYwRqIDpjTMhGsgNENuwjtrua7HgwOrLpILFth+PN9ib0v0/ucS/CQOoWOpr9RTYeILDYc2+F1+3GqW1BtXsXuMnq3tqdjFQQ5sQcCn58CXUffIT6j/+O6BuHcY+1EnifN9HJLMgi6z3zcepacGpaEiXu0XcOQcxBgj4IWihHgatQbXErw5DEUBIABUjQh+QEPb9iU7v3ZlbkYe+qQ0zBmBDCKM3yJku9U4tqjSIFYyOIptEkm8hruwHwXX1uUvc1i3LAHFmHiIgQWDzVm1nduQBUKSTLT2B26YAv/kLL53mtMQbY0tu/cDKxPZ2sDdf7bZKAhX/+JIyckXFdZ6SCAAhdMo+sa0+j7cdvYFYWUvTcxwiu6DryzyzMwSw8njecc/kZiYrGyPO7Uc1HQQmqNT6pqiALsQwvh1l5bxAdFq1S4Li4Te3gM7EWlmL4j//3KtcFAbclhqEVhGYcouI/qtaM5M8R6S8wnCz8c8rwz+ma9RhYNKXPDKPe8M0oYTDREt+04i5WV/DMvseXJouMVRAABfdeRPD9lQQvmu1ZBf3QudzdnF4A24+iog6qLYYEzETrXm+dgNn5uQbGhGxUOIZqj6Ka2nF8JkZuEDEM7xbyJZSNRjPeSLhDkhxMHW8NNlNJRgapO5Cgj9Cl8wekHLpjTs7z7sTTWCVrAApGBCPk93yVWQFAvHhFx/kcH6o1pmdfa8YlKj7bwD5Qh9sSSd6+zWGafv4ysb1jd3RtupLRCmI4iN9CTAPVHkNFnAEpiMRzRTCy/Jj5IUQE5bioqO0FuF2FCvdMk9VoMh0jL0jgtGnYB+q8rJ8k4bZFwdbJHyOBVhB9YRm4jfEmcINQEN1xWyO4zWEkFJ8W1TI+3EyrV6/m/PPPH20xNGmCBHz447MN3KbkpWV2WCYj0cl1vKMVRB+IZXipSgxPQRjZARBvBjY+A9Xae+1FMlm9ejUigojg8/koLi7mnHPO4c4776S1tXXEX388ISIXisg2EdkpIrf2cn6aiPxVRDaKyN9EZPJoyDnaODXNOEcakZwATmN70vZ145mFI53mOh7RCqIvOoLSfmNY5quYBkZuCFwXozSUskD1ueeey+HDh9m3bx8vvPACq1at4r777uO0004b1CAfzYkRERO4H7gIOAn4iIh0bx/7beBhpdQi4L+B1LTHTTOiWw/T/tIOjLyQ1xYjSai2iJfi6s/onJtRQSuIvojnVSfDdBWfieQEEZ8BfiMxD3sk8fv9lJWVMWnSJBYuXMgNN9zA2rVrqa6u5tZbu17o3nvvvcybN49gMEhlZSVf//rXE3MZ/uM//oO5c+f22P+GG27gnHPOGbA8Sim+/e1vM3PmTPx+P7NmzeLuu+/usuYPf/gDp556KllZWRQUFLB06VI2bNgAQCwW43Of+xyTJ08mEAhQXl7OypUrB/vfkmyWAjuVUruVUlHgEeCD3dacBDwfv/9CL+fHBW58oI+R7ymIZCVrmEW5+OeV60ymEUCr3D4QS8Dt6V5q/dPGHmt904vxz5+Esh3a/ry55/nZE/FXTsRpCRN+cy/yVnzf+IfaP7cc38wS3JYI7X/f1uW52RctStrfVFFRwapVq3j44Yf5yU9+gmEYfOUrX+FnP/sZd999N4sXL2br1q1cf/31hMNhvvrVr3LVVVdxxx138Oqrr3LmmWcCEIlEePTRR/nmN7854Nd+4IEHuP3227nnnnt4z3vew1//+lc+85nPkJubyzXXXMORI0e44oor+NrXvsYVV1xBOBxmw4YNiRGk9957L7/+9a/5v//7P2bOnMnRo0d56aWXkvZ/M0QqgAOdHh8Ezuy25i3gMuAe4FIgV0SKlFI9eydkMKo9ipEbInDKVIKnTkvaD7pvZgm+mT2bbmqGj1YQfeE3EWUi+cmbcys5QSRoeTnh7bYXuE7xlc+CBQtoamqipqaGnJwc7rzzTn77299y4YUXAjBjxgy+9rWv8e///u989atfZc6cOZx55pk8/PDDCQXx5JNP0t7ezpVXXjng1/3mN7/JTTfdxHXXXQdAZWUl27Zt4+tf/zrXXHMNhw8fJhaLceWVVzJ9+nSALq249+3bx5w5czjvvPMQEaZOnZoYJJTmfB64T0RWAy8CVYDTfZGIXAdcBzB1au8dQ8cyqi2KTMzv0ehu2PvGbLBMbUGMAFpB9IGYBr75E3sc7+uKXiyzz/NG0Ef2P5+CW9eCWxfGmJCFWXK8Q6ORE0iqxdAbHaa9iLB582ba29u5/PLLu3zBHMchHA5TXV1NSUkJV111Fbfffjt33303Pp+Phx9+mEsuuYSCgoEVPTU1NXHw4EHe/e53dzl+3nnncc8999DW1saiRYu44IILOPnkk1mxYgXLly/nsssuY8oUr0r16quvZsWKFcyePZsVK1awYsUKLr744h6zsFNMFdC5jHZy/FgCpdQhPAsCEckBLldKNXTfSCn1IPAgePMgRkrg0UA5LipiY4T8KNslsnE/1sT8AfUu6o/mR17DP7eM4NKZSZBU0xkdgxgFxBAwDCRkjUrK6+bNm8nPz6eoqAg33v7gscce480330zcNm3axI4dOygs9JqArVy5kubmZv74xz9SXV3NM888w1VXXZVUuUzT5E9/+hPPP/88Z5xxBo8//jhz5szhqaeeAmDx4sXs2bOHb3/72/j9fj796U+zePFimpqSl1M/BNYBlSIyQ0T8wErgic4LRKRYRDq+a18EfppiGUcfEbIvORVf5UQwhejmKm+O8jBRMdurgUiyVaLx0ApilBCfCYHUK4iqqip+8YtfcNlll2EYBgsWLCAYDLJ7925mz57d42aaXvbWhAkTuPjii/n5z3/Or371KwoLC7ngggsG/Lp5eXlMnjyZF198scvxNWvWMGPGDLKyvIEnIsLSpUu57bbbePHFFznvvPP42c9+llifk5PDpZdeyve+9z3Wr1/P1q1bWbNmTRL+Z4aGUsoGbgSeBbYCv1ZKbRaR/xaRS+LLlgPbRGQ7MBH4+qgIO4qIIZhFORjZ3pyUZGUyuW06xXUk0S6m0cJnIpEYrqtGrAV4NBrlyJEjuK5LbW0t//jHP/jGN75BaWkp3/iGl2mZk5PDbbfdxm233YaIcP7552PbNps2bWLDhg1861vfSuz38Y9/nCuuuIKtW7eyatWqhPIYKF/84he5+eabqaysZPny5Tz//PN8//vf5/777wfg5Zdf5q9//Svve9/7KC8vZ8eOHWzcuJFrrrkGgP/5n/9h0qRJLF68mKysLH71q19hmiZz5sxJ0v/Y0FBKPQ083e3Ylzvd/w3wm1TLlU449a04R5vwzSpBfBZGfginpmXY++oiuZElJQpCRH4KfAA4ppQ6uZfzgpfh8X6gDVitlHojFbKNFuIz4+3CTVRrDCnwfmyT2Xjs73//O+Xl5ZimSX5+PvPnz+fGG2/kU5/6FNnZx+Met99+O+Xl5dx3333cfPPNhEIh5syZw+rVq7vsd9FFF5Gfn8/WrVv51a9+NWh5brjhBlpbW7njjjv45Cc/yZQpU/jmN7+ZUAD5+fmsXbuW+++/n/r6esrKyli1ahW333474Fkh3/3ud9mxYweu6zJ//nwef/zxXlNwNemFfbiByKu7saYXIz4w8rOw99agHLfLPJXBohXEyCKpaBwnIu8GWvCKhXpTEO8HbsJTEGcC9yiluqcK9mCkB7v3NdB+uCilUK0RnL2NyIQQVkUezrEWnMMt+BZO7NJZVjNy9PUeD3Sw+0gw0p/tVBNev4fo5ipyP342IkJs1zHaX95BzgdPw8gbevt7p6aF2L4aAoum6F5Mg2Cgn+2UWBBKqRdFZHofSz6IpzwU8IqIFIhIuVLqcCrkGw1EBMkJ4hjN3qQ5V+EcboGY63WPzdFXRJrMQbVFkazjI0GtGSXkziwZtrVsFudgFuf0v1AzJNIlSN1bsVFFbwtF5DoRWS8i66ur+5/nms4opZBcPyps49a3Q8zLKNIzIzSZhtsW7VL/IIYkxZXqtkZS0pVgvJIuCmLAKKUeVEotUUotKSkZ49WTjkJCBhI0cQ42eUOJ/Ma46faqGT+oeJuNzoTX7Sby1v5h7du+5h3ant8yrD00JyZdspj6LTbKSExvoJAETVRzDGNqHqo5mpJurxpNKsl6/yld5zjjxQ9wXAKnDL1q3G2LYpbkDlc8zQlIFwviCeDj4rEMaEyX+MNIBvFFBPwmBC0wBaMoC8nxe2NOoz06MWiSjJ7slzqMoA8jq2vLGiM/hDOMpn1KKVRbVNdAjCCpSnP9FV6xULGIHAT+E7yZ3UqpH+DlkL8f2ImX5np1KuTqD5/PR3t7e6KIayQQywTTxpySh5iGN3UOLw4h/qFnd2j6p729HZ9v6HM+NAPDbYsSfecQvlmlmPnHv0tGfhZEbVQkhgSH8CMftcFx4+N9NSNBqrKYPtLPeQV8KhWyDIbS0lKqqqqoqKggFAqNSDOwRD1EPGtJsiwQcFtjGBO0ghgJlFK0t7dTVVXFxIk9e21pkovb1E70rQNYZfnQWUHE01vdxnaMISiIRBW1brMxYqRLDCItycvLA+DQoUPEYiMXF1C2i8TjEQBOYzM0CWazTt8bKXw+HxMnTky8x5qRQ7V1jO3t+kNuFmRh5IfAdnt7Wr9IyEfwXbN0DGIE0QqiH/Ly8lL+I9Lwgz/R9uM3KG+81XNBaTRjmONX+t1iELlBci4beh2iEfTjnzdpWLJp+iZdgtTjGrc1QuTN/bgt3pWWf9lkVFuM2NvHRlkyjWb4qLZofCRoci923OZ2nDo9X30k0QoiDVBRm8iGfdiH6wHwLSwFwH6nZjTF0miSggrHulRRdyby5j5an9k0pH0jm6t6ne6oSR7axZQGGAVZSMDCOdIIlWVIfhBAF8xpMoLguXNOGGdQjsI52ohy1aD7j+kU15FHWxBpgIhgluVjH2kEwIhnNGkFockEROSEjfSMvBC4Crc5POh9O/o7aUYOrSDSBLMsH9USwW0JJ1Je3WatIDRjG6UU7S/tIHag9+lxRn481bWpbdB7d+/vpEk+WkGkCVZZPpgGbmN7fNqcqS0Izdgn5hDbfgS3oXcFYOYfr4UYDEopVLu2IEYaHYNIE4wJ2eSueldieIqRG0A1R0ZZKo1mePQ3ElQCPqxpRYO3BBSE3jMfIzc4XBE1faAVRJogIl7zvo7HOX5cbUFoxjgDmfiW9d6TBr2vGIJvatGQ5dIMDO1iSiPsQ/W0PvUmbnvUa9qnFYRmjOO2D2wkqHIH17DPbYkQO1CHiummliOJVhDphAhOdTNuXStGrh+lg9SasU7MAcvoMx01uu0wzT9/aVCDf+xD9bT/ZTMqolvjjyRaQaQRZmE2AE5ti3YxjSFE5EIR2SYiO0Xk1l7OTxWRF0Rkg4hsjM9gHxf455WT+9GzEN+JvdkS9IOrcAaRyZRwXeksphFFK4g0QgI+JCfgKQgdpB4TiIgJ3A9cBJwEfEREujvVvwT8Wil1KrASeCC1Uo5tn3b/AAAgAElEQVQu/XVBNoaQyeS2RZCAlUjq0IwMOkidZpiFObh1rToGMXZYCuxUSu0GEJFHgA8CnedgKqCj42M+cCilEo4i4Vd3ISE/gUVTTrjGyA2CgHOsCWZ77dfdpnZUtJvLyTCOW9nVzTrFNQVoBZFmWJMKsMWrptYKYkxQARzo9PggcGa3NV8BnhORm4Bs4PzUiDb6xPbXYZX23Y5bTAMjPwt7bw2cVQlAeN0e7P21XdflBMi9Yqn3wHET8yQ0I4dWEGmGf/4k/PMnEXuuSldSZw4fAR5SSn1HRN4F/FxETlZKdWlQJCLXAdcBTJ069DnN6YJXzBZBsor7XZt1/gLcpuMupsApU/DN6TrMqbM7KXjOHIwcXQMx0mgFkaZIjh+iDipqI379NqUxVUBn/8nk+LHOXANcCKCUWisiQaAY6NLPXSn1IPAgwJIlS8b+wOyoDY4akCvIyA12KXozi3Ppqzm4VaoHPaUCHeFJQ1qe3AAl3tdDu5nSnnVApYjMEBE/XhD6iW5r9gP/BCAi84EgUJ1SKUeB/qqoNemPVhBpiAR84PMyP3Sqa3qjlLKBG4Fnga142UqbReS/ReSS+LKbgU+IyFvAr4DV8TnsmY3jYuSHkOxA/2s1aYn2XaQhZmE2TlU9WKItiDGAUupp4Olux77c6f4W4OxUyzXamMW5wxopqhl9tAWRhphFOQAYk3J0NbVGoxk1tIJIQzoUhDktR7uYNGOWyFv7afvrlv4XatIWrSDSEMkNYhbn4x5u09XUmjGLU9PSJXVVM/bQCiINEREC8ypwdjbqGIRmzKIH+ox9tIJIV7J9GJOycVu0BaEZm7htUZ3iOsbRCiJNceqayP7askQuuUYzllBKodqiSJZOcR3LaAWRpphl+d4de+A98jWatMF2McvyMQuyRlsSzTDQdRBpilmYjYq5gJ6YpRl7iM8k+8KFoy2GZphoCyJNEcPAPdqWqKjWaDSaVKMVRBqjqsOQZTIeujJoMovozqO0/GZdYia1ZmySMgWhxzIOHmd7I8762v4XajRphmoJ4zaHdSfiMU5KFIQeyzhE2lycnU39jmzUaNINty2KBH16JOgYJ1XvXmIso1IqCnSMZezMuB3LeCIkN4DkmdhHm0ZbFI1mUHgprroGYqyTKgXR21jGim5rvgJ8VEQO4nXGvCk1oqUvRo4f65wyopsO9L9Yo0kj3PYoEtIKYqyTTvZfx1jGycD78cYy9pBPRK4TkfUisr66OrNnrkiOH7eqBae2ZbRF0WgGhTVpgp76lgGkSkEMdCzjr8Eby4g3davHMFul1INKqSVKqSUlJSUjJG56ILl+nD3NqLYoblhng2jGDsHTpxNYPPbnao93UqUg9FjGIWDk+HF2NgDg1raOsjQazcBwm8MoV6dmZwIpURB6LOPQkBw/zl4vQB15bT9NX/rrKEuk0fSNUorWP22k/e/bRlsUTRJIWZKyHss4eCQ3AG02wXPn0vwffyP86y2EVi3CNz+zXWuasYtb34pqjWCVF4y2KJokkE5Bak03JMfLAjEMi9jrhwFo/8XG0RRJo+kTe38dANaUwlGWRJMMtIJIY4xcT0HEdtdiLSmGLIu2X27SrTc0aUvsQC1mSS6GTnHNCLSCSGM6LIjYhsP4L5hK1tWLcPY0EF2r6yKSiVKKH/3oR7z3ve9l0aJFALz44osAEwby/AG0kblLRN6M37aLSENS/4A0wW2N4Na0YE0tGm1RNElCK4g0pkNBRP+2H4DgZXMgaNH+i02jKVbG8eUvf5mf/OQnXHfddezf7/1fT548GaCsv+cOpI2MUuqzSqnFSqnFwL3Ab5P8J6QFEvKRdcHJ+GaWjrYomiShFUQaY+R607iia/bhVrejXIfQJXNp//VmVEzPiUgWDz30EE899RQrV65M9L2aMWMGwEDGoQ2kjUxnPoKXpZdxiGFgTZqAkaOnyGUKWkGkMR0WhGqNQbuLU9NCaOXJuDVtRF85OMrSZQ6O45CTkwOQUBAtLS0A7gCePpA2Mh17TwNmAM+f4PyY7RKgYg7h9Xtwm9pHWxRNEtEKIo2R3OOBPiMnCEphzvBGkTqHm0dLrIzjoosu4nOf+xyRSATwYhK33347QLJjBSuB3yilejX/xnKXAPtQPdFNB3FbI6MtiiaJDFhBiMh7RGRG/H65iPyviPxMRPr102qGhpFzXEFYsyeSs/JMzHKvv417TFdWJ4u77rqLw4cPk5+fT2NjIzk5Oezbtw88a6A/BtJGpoOVZKh7yd5fB34Lc6Luv5RJDKZQ7gHggvj978T/bQceBC7p9Rma4RGwwBRwFP7TJiEiGEUhEHCr20ZbuoxAKUVNTQ2PPfYYdXV17Nu3jylTplBWVoaIDMTFlGgjg6cYVgL/0n2RiMzDy4pam9Q/IA1QrsI+UItv8gTE0E6JTGIwCqJCKbVfRCw8RTENiKLnNowYIuLNhLAMzMl5hN/Yi2qLYhSGcKu1BZEMRISFCxfS3NxMaWkppaWDy8BRStki0tFGxgR+2tFGBlivlOroObYSeCQT28c41U2oiI01Rae3ZhqDURBNIjIROBnYopRqiTfe842MaBrwiuWs+SWICCocI7avFqMkG0e7mJLGqaeeyvbt25k3b96Qnt9fG5n4468MWcA0x22JIAELa/KAykY0Y4jBKIh78cxpP/CZ+LGzgXeSLZTmOHl3rsCc4gWmzeJcYtuOYM4u0C6mJLJ8+XIuvPBCVq9ezZQpUzqPeNWXxAPAP6sU34wSxNCjcTONASsIpdS3ROR3gKOU2hU/XAVcOyKSaQDIWrkwcd8szgXAmJFH7C/7R0ukjOOll15ixowZrFmzpvsprSD6QbkKMUQrhwxlUN1clVLbO+6LyHsAVynV41ulGRmMgiywDMzyLCLaxZQ0XnjhhV6Pi8j2Xk9oEkS3VBHbcZTsDyxGfOZoi6NJMgNWECKyBrhNKfWSiNwCfA6wReR+pdQdIyahJoEYgm9GCbHqQ7h17SjHRUydNZIM6uvrefLJJ6mqqqKiooKLL754tEUaE9gH6kBEK4cMZTC/LicDr8TvfwJ4D7AMuD7ZQmlOTOicORj+EChwa3UcIhmsXbuWWbNm8YMf/ICNGzfywx/+kFmzZgFkj7Zs6Y5T16prHzKYwbiYDECJyCxA4gN+EBGdupBipCQLTME91opZmjPa4ox5PvOZz/DAAw+wcuXKxLFHH32UlStX6qHKfaBcBVEbCepExkxlMAriH8B9QDnwO4C4sqgZAbk0J8Bti+KEG/GdU57IZIptPkbkr7uxTirBt7gMs1hf+A6G7du3c+WVV3Y59uEPfxgG1qxv3KKiMQCMgFYQmcpgXEyr8XrTbAS+Ej82D7gnuSJp+kJCPhDBmJ6HEy+Wa7r9eRo//Qy1K37OkUnfwT7QOMpSji0qKyt55JFHuhx77LHHAHRjoX7wzSnDKNIXJJnKYNJca4Hbuh37Y9Il0vSJiGBOyMackZfox2RvrSHwvlkEL5lL441PY2+vxYrXTmj65+677+YDH/gA3/ve95g2bRp79+5lx44dADqXuA+MoJ/Q2ZWjLYZmBBlMsz6fiPyXiOwWkXD83/+KV1NrUohZno9RkY1b04aKOdi76vCdVk5gxUwA3EO60+tgOOuss9i1axc33ngjp59+OjfddBM7d+4E0LnEfaBcV4+/zXAGE4O4E284yvXAPrxeTLcDecBnky+a5kSYpXmIaeC2tWHvqYeYizW3CLPcK6RztIIYFFVVVWRlZfHRj340cay+vh50G5k+ie08RvilHeRccYbXjl6TcQwmBnEFcIlS6jml1Dal1HPApcCV/TxPk2TMklxirxzFOdKKva0WAN+8YozcAJLr17MiBsmHPvQhDh7s2tk7/njWqAg0RlARL0gtOkidsQzGgjhRLb2usU8xRsiP+46nBOx3vCQya24xAGZ5rnYxDZLt27ezcOHCLsfij/VlcR+osO21o7d0sWamMph39jHgSRG5QETmi8iFwO/jxzUpxijPQZku9js1GKXZGBNC3vFJudrFNEhKSko6Yg4J4o/14O8+UJEYEvB1bm6oyTAGoyD+f+AvwP3A63jdXV8AvjACcmn6wVxQQPBjc7D31GHNKz5+fFIuzuGWUZRs7PGv//qvXH755Tz11FNs2bKFJ598kssvvxxgbA2GTjEqHNPupQynTxeTiLy326G/xW8CdKQvnMMJhrBrRg7D8r6YblsE/ykVieNm3IJQSukruwFy66234vP5+PznP8+BAweYOnUq1157LZ///OePjLZs6Yw1rQhi2sjKZPqLQfzkBMc7lEOHopiZNIk0A0LyQiiiGCVBrLnHu1Ib5TkQtlENYSTudtL0zZo1a/jwhz/MF77wBQ4fPswtt9zC22+/DYPsdjze8FfqcfSZTp9fAKXUjFQJohkcZkk2ka21mAsK8XVzMQE4h5sTcQlN33zyk5/k2WefBeDmm28GwOfzAUwfNaHGAG57FPFbuqNwBqPf2TGKWZqN/WYNZmUB5qyC48cn6VqIwVJVVcXUqVOxbZtnnnmGBx98kO9///ugu7meEOUqWh59lchbutg8k9Em9BjFKMkm9vdD2FvryL3q7MTxjmI5neo6cPLy8jh69Chvv/02CxYsICcnh2g0CjqF+4SoqA1K10BkOimzIETkQhHZJiI7ReTWE6y5UkS2iMhmEfllqmQbixglWajaMIbPj2EdH9ZilHvtv7UFMXBuuukmzjjjDFatWsWnPvUpwBtDCoQH8vzx+NlOFMnpVt8ZTUosCBEx8dJjVwAHgXUi8kTHTIn4mkrgi8DZSql6ESlNhWxjFaMoCwSsJWWEX9lF4IwZiGlg5ASQvIBOdR0Et9xyC5deeimmaXYMCqKiogK8ljJ9Ml4/2ypsA2AEtBMik0mVBbEU2KmU2q2UigKPAB/stuYTwP1KqXoApdSxFMk2JhHTIHTFAgLvnUZ06yGcww2Jc6Yulhs0c+bMSSiHjsdA+wCeOi4/27rNxvggVQqiAjjQ6fHB+LHOzAHmiMhLIvJKvFJb0weFj15B9kdOBcsgdqAucdwoz9ExiNQxLj/bRn6IwOnTkVzdjSSTSacsJguoBJYDHwF+JCIF3ReJyHUisl5E1ldX60JXsQysSROw99cmWi971dRaQaQRGffZNvOzCCyagqFjEBlNqhREFTCl0+PJ8WOdOQg8oZSKKaX2ANvxvlRdUEo9qJRaopRaUlJSMmICjyWsqUWotihurTe+oHM1tWbEGZefbbc1gtsyoBi+ZgyTKgWxDqgUkRnxAUMrgSe6rfk93hUWIlKMZ5bvTpF8YxprygQk5MNt9b6wZnkuRBxU/UBc6JphMi4/25E399P61JujLYZmhEmJglBK2cCNwLPAVuDXSqnNIvLfInJJfNmzQK2IbCHeBDA+5lTTD0bQT87/dya+aV5FtdFPsVz777fiNumrv2QwXj/bulHf+CBlOWpKqaeBp7sd+3Kn+wr4XPymGSQi4rmUlOrUbqMF38kTu6yzDzRSd+mj5N97ETk3njkaomYc4/GzrSIxXQMxDkinILVmGLjhGC2/WUds25E+2204B5u8f/c3plQ+TWahIjaiayAyHq0gMgQJWIjhpbt2VFP3luraccyp0llOmqGjXUzjA30JkCGICNaUQqJbDyGWgRQEcaqaeqzrsCo6LAmNZigEl85EcgKjLYZmhNEWRAZhTS0CV2EfqsesyO3VSkgoiF6Uh0YzUHyzSrEm5o+2GJoRRiuIDMIszUMCFvb+OsyKvL4tiCpdJ6EZGirmYB9tREXs0RZFM8JoBZFBiCEETp+ONbPkhBZEIi4RtnHrdJ2EZvA4Da20Pb0R+5i2QjMdHYPIMPxzywEwK/Jwj7SgbAfp1A7cOdQMlgG2i1vVhFmUNVqiasYoHZ1cJah/PjIdbUFkIG5LGJkaAlfhHm3tcs451IzvFK82QmcyaYZCRydXQ2cxZTxaQWQgsd3VKMvGmNE1DuG2RVENYfxneM1GdSaTZih0xB50mmvmoxVEBuKfPwlMg8AlM7pYCW58iJDv9HIQncmkGRoqHPOGsfrNftdqxjbaiZiBiM/EN2siOC52p0FCHRlM5rQCjIk52sWkGRK+WaWYxTmI6JHdmY62IDKUwJKpqJYYrhNJHEsoiEm58Syn5FsQblNYp89mOGZBVqIxpCaz0QoiQzECfmIvH0E1RVGu94PtdlEQeUmPQSjb4cj0u2n++otJ3VeTXtiHG3Bq9Mzz8YBWEBmM2teG/fxhxPBcAc6hZghaSEEQc3Je0l1MbmMEVR+m5btrcVsi/T9BMyYJv7qLyFv7R1sMTQrQCiKDMeLV1E51M05DG86hZsxJuYgIZkUuqq4d1R5L2uupRm/GhKoP0/bTDUnbV5NeqLDu5Dpe0AoigzErcnGOtdL63NtEXt+bUBDgKQ9IbiaT2xAfQhQwabnrFZTtJG1vTXqglNKzIMYRWkFkMGZFHqo+jH9WKfb+WpRtJxSEWRGfGZFEN5Nq9NxK2TecgbO3gfbfbEna3po0wXbAVboGYpygFUQG06EEjOxs8JtYpxclxpGak0fAgoi7mLJWLcSsLKT1wdeTtrcmPdBtNsYX+l3OYMy4G8k90urVRUQdaJAu55JpQbhxC8KYEMJ/ajmxt44kbW9NeiAhH1nvX4SRGxptUTQpQFsQGcxxN1ITZl4uzsEWjKIgAEZuAMn1JzXVtSNILQVBpCCYUBiazEEsE2tiPkaWf7RF0aQArSAymIQFUdWMOtpK23++im9K0fHzk/Nwdtcn7fU6gtRGXgAjP3A8aK3JGJyGNqK7jukEhHGCVhAZjIR8SGEIp6qJ6MsHQIGU5xDddQynoY3AhbMJP70jaa4g1RhBsnyIz8QoCELYHhdDZUTkQhHZJiI7ReTWXs6vFpFqEXkzfrt2NORMBnZVPeEXt6Fsd7RF0aQArSAyHLMil9afvUnTl57Hf85UrCm5hF/ZRfjlHeR+6d0YE4I0/PufktIew20MI/nenGLJDyaOZTIiYgL3AxcBJwEfEZGTeln6qFJqcfz245QKmURUxGvUJ34dvhwPaAWR4fjmeT1z8r79PopfuAojO0hw6Qyco004xxrJ+/o/EX1xH+2PbR72a7mNEYy4YjAK4goi891MS4GdSqndSqko8AjwwVGWacRQERvxW4nqfE1moxVEhlPw4MWU7f8suTeflZgs55s9EbM8n/D6PQT/ZQG+xWU0feHPKHd4bgPVyYLoUBAq8xVEBXCg0+OD8WPduVxENorIb0RkSm8bich1IrJeRNZXV1ePhKzDRoV1kdx4QiuIDMcoCGGWZHc5JiKEzqoEVxFZt4es607H2d94fF71EHEbIwnF0KEodCYTAE8C05VSi4A/A//b2yKl1INKqSVKqSUlJSUpFXCgqEhMF8mNI7QjcZxi5IUILp2JhHw4MW9mhL2nAXNy/pD3VA1hjOkF3v7jx4KoAjpbBJPjxxIopWo7PfwxcGcK5BoRQufMRTk6QD1e0BbEOMY/rxzftGLMGQUQsrCHmfLq9uJiGgcxiHVApYjMEBE/sBJ4ovMCESnv9PASYGsK5UsqRk4AM18XyY0XtILQoJwY2d94F87B4SqI40HqgWQxuY1hIi+N7bbRSikbuBF4Fu+H/9dKqc0i8t8ickl82b+LyGYReQv4d2D16Eg7fCKbDuJU60mE4wXtYtJgVRSAAlVi4NS2YBblDHoPFbUhbB9Pc83xgyF9uphavruW5jv+Tnn9LRg5gSHLP9oopZ4Gnu527Mud7n8R+GKq5Uo2ynaIrN8Dp0/HLMkdbXE0KUBbEBqMvBCx56og4tD6zKYhTQtL9GHqsCBE+m23EdtwGGwXZ1/j0ATXpBQV9maH6Cym8YNWEBoArNJc2n+4GfGbtD67Cbc92uW8UoroawdRsd5bLHRYCh2xB6DfdhuxTccAcPY1DFd8TQroqIrXw4LGDylTEP21I+i07nIRUSKyJFWyacCcMQFnSy1Z7z2J4OnTMUJdm7FF/7aX6jN/zLGFDxB+enuPyuuOWEOHiwk8ZXEiF5PbHMHZG8+e0hbEmCBhQeg013FDShTEQNsRiEgu8Gng1VTIpTmONcOLQ7i1YfzzvKQb+0gjsd1ewVbkH/tBQLmK2n/+Ja0PrOvy/O4uJiDuYupdQdibjyXudygKTXqjItrFNN5IlQUx0HYEXwW+BWR8bmS6Yc6cANClu2t0cxXta94hvGEfsXVVWHOLmfj2JzFnFxL5y+4uz1e9WRD5wRO6mDrcS5Ll0y6mMYI1rZicK5di5AX7X6zJCFKlIPptRyAipwFTlFJ/7GujsdCOYCxizfAUhL3n+I91aPk8fJUTib65H6nMxnfmJMRv4Vs0Efudmi7P782C6MvFFNt0FMn24V82GVtbEGMCMQ2M7ABi6NDleCEt3mkRMYDvAjf3t3YstCMYixjlORAwcfYctyDENAieXYl/TjnmSROwzilFRWJY84qxd9Z1CVh3WBCdg9R9ZTHFNh3DOrkUc0aBzmIaI8T21hB5++Boi6FJIalSEP21I8gFTgb+JiJ7gWXAEzpQnTrEMLCmFXSxIMBLV1XHYrTf8xZmYTb4La9DrO1i76pLrOtwJUleZxdTANUUSbRmCD+9Hbc5glIKe9NRfAsnYk4rwD3SkgiAatKX2N4aYu8cHm0xNCkkVQqiz3YESqlGpVSxUmq6Umo68ApwiVJqfYrk0+DFIXqbMBddV4WztZ6s95+MiGDMzMe3YgqxrcddfG5jBMnxI+bxj1SiH1NzBPtAI7X//EsaP/cs7pEW3Np2fAtLseK9m+z92opId1REd3Idb6REQQywHYFmlLFmFGDv6akgYusO4VtYmvhxUIZN8CNzsI/UJjJbOrf67kA69WOyt3v96tp+uoH2R98GiFsQXnNA7WZKf1RYd3Idb6QsBqGUelopNUcpNUsp9fX4sS8rpZ7oZe1ybT2kHnPGBFR9GLehPXFMKUV0/SF8ZxzPKQgum0Xkj/tQfkXLExtwapq79GHqoOOx2xA+bpn4DBpv/QuAF4OIWxA61TX9URFbF8mNM9IiSK1JD6yZPTOZnJ11qIYw/jMmJY6JCNLgEP3DPlDQ+se3ICQY3SyIhIupMeLFK3wGeV9ZDhEHY2I2Zkk25qRcMCWR6tpy11oav/Ac4b/uHhfzrMcSKmJrF9M4QysITQJzlqcgYhuOByKj67xcAv8ZXYekWfOKib14kOyLF+ObW4azqxEpCKJix3/UE0ODGsLYu+qxZkwg5zPLMKcX4FviKRyxTMwp+dh7G3AONdH4+edo+fbL1J7/MNVn/SQps7I1ySF31bsInDZ9tMXQpBCtIDQJfIsmYi0ooeWuVxLjR8NPbkeyfFgLuqYUW/OKUY0RVGOE0LLZuDXtGAVBWv+wgbY17+C2hLvMhHB212POnIAEfZSsvYYJD30osZc5LR9nXyNtP98IrqLkjX8j55azib1xWLue0ggxBLH0T8Z4Qr/bmgRiGOTeeg7228cIP7WdyD/20f7I22R/ZllinnUH1rxigETBXEeQ2ppRgr2vlpbfvk7scC0ETFRDGHtXHVbcQjHLcjGLj49BNacV4OxtoO1nG/CfPQX/qeWErlwAQPRVnXefDrjNYdpf3oFT3zraomhSiFYQmi6EVp6MOb2A5q//ncab/oQ5JY/c287tsc7XTUG4jRGMvCDB06eTc9npWNOKiG0/QvZ/n4l9oAHVGEm08+iONb0A52AT9rZasq4+1dt/4UQIWkRfrer1OZrU4jaHiW07ksha04wPtILQdEEsk9xbzib2WhWxN4+Q/90LMLL9PdYZFXlIjp/YOzVekVvUSQSpjZwgWefNI+uiRTi7GomtPwSANauw19fsSHWVLF/CchCfif/0cmLagkgLEo36dJrruEIrCE0PslYvxpyaT+B9swhe3qPpLuBlMlnzirG31iTaaUi3NFerLJ/Y0weIvX4EyfVhTu59Ul1HsVzwwydh5B7PhPKfOZnoG4e9aXWaUUUriPGJVhCaHkjQR+lb11P01L8gIidc51s0kejLB4i97lkInfswJfbKD6BaogRvXERk9yHctp69mXynluNbWkHOZ5Z1PX5mBUQcYhuPDvMv0gwXFdbDgsYjWkFoesUoCCE+s881uV8+D3wG9R//HUCPSmpvH09pxP5ShQrbtD2zCbet67Q6Y0KI0lc/gf/U8i7H/WdOBtBxiDRAuS4SsLq0UtFkPvrd1gwZa1oBEx6+FLfWq7zuXkkNx9ttGD4fWe9bgNsaoe2ZjT1GmvaGOTUfY2J22mcyjYdpicHTppPzkWX9L9RkFNpe1AyL0AfmknPL2bR86yXM8p4xhg6lYc6agDUxn6wVC2h7bjPhV3eTtXxen3uLCP4zJxNLYwui07TEFXhzTtaJyBNKqS3d1vU5LfGNN964wLKs/1RKlQHGnXfeydatW0dYek0m4vP5KC0tJS8vb9h7aQWhGTZ5d/wTWVctxppd1ONch4upo42HVVZA9vtPAcOLbbjN7US3H8W/oAKjlzYO/jMrCD+xDbe+HWNCKCnytj60gbYfv0HRnz7aJSg+RBLTEgFEpGNa4pZu6zqmJX6h+wZvvPHGBYFA4L7p06dHQ6FQvWEYasuWLdPmz58/XNmShtsaAUN6zCrXpBdKKdrb26mq8i6qhqsktItJM2zEMPDN7314U4eLqXOKq1mc482WAOyqBqIbD9Dy2GuEX9uN220uhG+p1+IjGg+EJ4PYG4eJvXUUyUnKj92wpyValvWf06dPj2ZnZ7cbhtFnbxFlO7gt4R43ZTsnPN9xbrC4rZHje4RjMMR9NKlDRMjKyqKiooJjx471/4R+0BaEZkTpqI04UZGcf145Zlke0Y0HiW6pIrb7GKFz52JVeOt9CycCYG+uhvNnJUUme0cdVmVhnxlayaLTtMTVJ1qjlCoLheKDbq8AABUMSURBVEL1R44cKa6pqSkBcOOtTnpZ3Gvar/i9r7Jyu513FcpVmHn9W1/KdXEb272xon7L66vlxvWVAD79czFWCIVCxGLDL2rU77hmRLFmF3q9nOKV171hFmQTevdc/CdX0P63d4jtrUkoCKM0G6MoRGzz8K+GOrB31OI/fVL/CwfGYKYlApThTUvsPBDLMAxDlZWV1ZSVldUAbNmy5fTeXkx8FmZh7/UkAIbfgk7nB5IM0IGKOuC4CfefWZDdzzM06UqyLn60i0kzogQvnU/ZoZsxi7L6XWsW5pB9yakEz5wJgFPXitvUjrWg1LMgesGtb6fl7rUcW/Yjwn/a0e9rqKiNs7cBa07PeMkQSetpiUbIP+C4gYraYBigU1k1cbQFoRlRRKRHhXWf6zs1BQyv3YlT14LvvEm0fe8NlFJdrozaH99C/cd/h2qLgSG0/XITwYsq+9zf3tMAjsKq7L3tx2BRStki0jEt0QR+2jEtEVjf20CsVKNcBY7bZ12LUgpijlfrkALXm2ZsoC8VNGlLaPk8zOJcjBlZhG5aSHRjVWI+RPuT26hb+Rt8p0ykZMO/Ebx4DrHX+k+HtXd4o0/NyqRZEGk/LdFtCeM2h/ucraFiDiiViGUMlNWrV3sXASL4fD6Ki4s555xzuPPOO2lt1Z1fxzpaQWjSFiM7QNaFC7GKJyB5fiJv7MHeX0vkhT3UffjX+E4to+iZj+JfXI5/aQX29lrc+vY+93R21AEk08WU9ojfAtf14gsnWiMCAQv6qZ7vjXPPPZfDhw+zb98+XnjhBVatWsV9993HaaedxtGj47dNSjQ68PhPuqIVhCatERECy2bQ+sW1SJuBNaWQptufx/9PU5jwfx/EyPPcV76Othzr+rYi7O21yITggGIimYL4vR/9vpoeis/EzA0Nyb3k9/spKytj0qRJLFy4kBtuuIG1a9dSXV3Nrf+vvXuPjqq6Fzj+/WWSTCAPSCAJihTkFUtu8UIQcIFAtVeQWqzWKhghtbaglUvR1VUULhVuQdR6W5CLCuECTS9ULpYKWgVBS7GKvASEJLxCghJQJLKAwEge87t/zGScJJPwyHMmv89as3Jy9vac/Ys77Oxz9uPJyhPL58+fzw033EBUVBQ9evRg9uzZlJV5yjVt2jRSUlKqXf/RRx9l8ODBNd5/w4YNDBs2jISEBNq0acPQoUPZtm1bpTzFxcVMnjyZTp064XQ66dKlC88884wv/eTJkzz00EMkJycTFRVFSkoKS5YsAWDTpk2ICMeOVZ7RHx4ezrJlywAoKChARFi+fDkjR44kOjqa6dOno6r8/Oc/p1u3brRq1YquXbsydepULl6svCbZxo0bueWWW2jdurUvhry8PDZt2oTD4eCzzz6rlD8rK4s2bdo0eC/NGgjT7DkSowmLb0XZrlO4TxRT8sFnOO/qiuuDg1x4Zx/lRcVE9rsWBEou8Zip7FAR4fX4eCkYSFgYhDs8o5QC0HI3Wkvv4mp07NiR9PR0Vq9e7RuyO2PGDF544QXmzJlDbm4u8+bNY+HChcycOROAjIwMDh48yNat30w2v3jxIitXrmTcuHE13qu4uJhf/OIXbNmyhQ8//JAePXowYsQIioo8jxNVlTvvvJO1a9cyf/58cnNzycrKIjHRM3fH5XIxdOhQ9uzZw/Lly8nJyWH+/Pm0bn3lf0RMmTKF9PR09u3bxyOPPIKqkpSUxIoVK8jNzWXu3LksXbq0UuO0ceNGhg8fTlpaGlu2bGHr1q2MGzeO0tJShg0bRo8ePXyNVYXMzEweeOABoqMbdqSZvaQ2QSE8NZGynC9x/dWz/ETU4BTUXcrFTz6jbO0uwrsmEjHg2ksuy1F26Csih3RujCLX2fm3P6l2LqJLeyK/fS1aVs6FDdnV07snE9kjGffXpbj+/s1SHVruBrebyF4dieyejLv4Iq73D1RKk3CHZ5Z7PUlNTeXs2bOcOnWKmJgYnn/+eVavXs2IESMAuP7665k1axaTJk3it7/9LT179mTAgAFkZWUxYMAAAN544w1cLhf33Xdfjfe5++67K32/aNEi/vKXv7Bu3TrS09N57733+Mc//sH27dvp18+zDFbXrl0ZMmQIACtWrCA/P5/Dhw9z3XXX+dKvxoQJE0hPT690bvbs2b7jLl26kJeXx0svveRrGGfOnMkdd9zB3LlzffluuOGbZWjGjx/PvHnzmD59OmFhYezfv59//vOfvPjii1dVxithPQgTFCJSkyjN/hLXqhzCUxOJTE3G+Z3riL33JiJ7d6LsaBERA6+lZFthjS9j1VVK+adn6m0EUzDx7Cft8PQmqnIriHg+9aji/4OIkJ2djcvl4kc/+hExMTG+z4QJEzhz5gxffukZxpyRkcHKlSt9k7yysrIYNWoUbdu2rfE++fn5jB07lu7duxMXF0dcXBxnzpzh6NGjAOzcuZP4+Hhf41DVzp076dWrl69xqIv+/ftXO5eZmcmAAQNITk4mJiaGp556yle2ivvffvvtNV4zIyODkydPsn79egAWL15MWloaffr0qXN5L8V6ECYohKcmomcvUrL5KLFPD/WdF2c4UWldiEztyPnzu3Cf3IHrvf2I00Hkv3SsNNmrLM/7gjpIHjFF39G7xjQJd9SaHhYVUXt6jJPoO3qj5W7cp88j0c56X2cpOzubNm3a0K5dO44cOQLAqlWr6NmzZ7W8CQmeRnv06NFMnjyZv/3tbwwaNIh169bx+uuv13qfO++8k/bt27NgwQI6depEZGQkgwcPrreXxGHeRtX/D4/y8vKAs92rPvJZtWoVjz32GM8++yxDhw4lLi6OVatWMW3atMu+f7t27bj33nvJzMzktttuIysri1mzZl1lNFfGGggTFCJSk3zHre6tvstdWFQEzv6evwDLT5xD3aWUHvqC8OsSPA1FhzaUtcARTP60rBx1lSLRTsQ7W7rixbXU8zIahYWFLF++nHvuuYewsDBSU1OJioriyJEjjBw5ssb/Lj4+nh/84Af86U9/4tNPPyUhIYHhw4fXmL+oqIicnBzeeustX75jx45VWocoLS2N06dPs2PHjoC9iLS0NJYsWcKxY8cC9iKSkjx17/jx43Tq5Jk0v3v37lqHDVfYvHkzffr04YknnvCdKygoqHb/d955h0mTJtV4nQkTJvDd736XhQsX4nK5GDNmzCXvXR/sEZMJCuGpnheK4SntCPdrLPxFfCcJnA7cu08Tc99NOPt8i/JT57iwbi8Xdx31zYFoiY+YAM86ThdLPWssVZwqKQNHGBJ+9f8UlJSU8Pnnn3P8+HH27t3Lyy+/zM0330xSUhJz5swBICYmhqlTpzJ16lQWLFjAgQMHyM7O5tVXX2XKlCmVrjdu3DjefPNNXnnlFdLT03E4ah56Gx8fT2JiIpmZmRw8eJAtW7YwZswYWrX6Zu2pW2+9lVtuuYX777+fNWvWkJ+fzwcffMDixYsBGDNmDJ07d2bUqFFs3LiR/Px83n33XVauXAlA9+7d6dy5MzNmzPA9/3/88ccva8RXSkoKe/fuZc2aNeTl5TFv3jxWr15dKc/06dN5++23mTx5Mp988gkHDhxg2bJlHDhwwJdn8ODBpKSk8Ktf/YrRo0cTGxt7yXvXC1UN2k9aWpqaluPkoMV6bu6W2vPcnKlfpL3i+95dWqYX9x/Xsq+K9auHX9fP+yzQCx8e0rKic5e8H56Z0A1ej3fv3l2gqjv8P9nZ2ZfzI7kibrdby06d0/Kzrm/OlZeru6Tsqq+ZkZGhgALqcDg0ISFBBw0apM8995wWFxdXy5+Zmak33nijOp1Obdu2rfbv319feumlSnlKSko0MTFRAd29e/cly7Bp0ybt3bu3Op1O7dmzp7722mvarVs3ffrpp315zp49qxMnTtQOHTpoRESEdunSRefMmeNLP3HihI4dO1bbtWunTqdTU1JSdOnSpb70jz76SPv27atRUVHau3dv3bx5szocDl+e/Px8BfT999+vFsv48eM1Pj5eY2NjdcyYMTp//nz1/NP7jXXr1unAgQM1KipK4+LidNiwYZqXl1cpz9y5cxXQbdu2XfJnoqqak5NTY9rl1m3Ry+gmNVf9+vXTHTsadVKqaeaKX/yIM79cR/sPH8Z5c6dKaScHZuLo1ZaIYddAuRLWPhZn705EdA78yElEdqpqg+/+tmfPnoIbb7zxlP+5nJyctF69qj9Kqyv3ua/RkjLCEqJtSY0g8+tf/5oNGzawa9euy8qfm5tLTXuKXG7dtkdMJqS0/mkfJD6K4hc+rHS+ZPcJSrcWEplyDbH3D8DZvyuUlaPnv26ikjaRSAeoQlk57gsl1fbfMM3PmTNn2L59O4sWLeLxxx9v1HtbA2FCSliMk+hHb+Lrv+b63jkAFP9+CxIdQfT4NMQZgTO1I9E/7EvEDfW27HdQkIhwcIR59o1wlUCpbQLU3N11110MGTKEu+++mwcffLBR720NhAk5Mf/eHyIcFP9hCwDlhWdx/XkfrR/uW2nbUhHxjeZpKSRMcMR7Hy+penoUplnbtGkTLpeLpUuX+obcNpZGG+YqIiOAeXiWRF6sqs9WSX8C+BlQBnwJ/FRVj1a7kDGX4OgQS+uxvTm/ZBdh18Ti/qIY3ErM5IFNXbRmo6GGt5rQ0ii1Q0QcwALg3/Ds2btdRNaqqv/G7ruAfqp6QUQeBZ4H7m+M8pnQEzfrVtxfnOfcb/4OQKsf9yL8+sDbnrY06lbPHtPQ4npQ5so01p8P/YHDqnoEQEReBe4CfA2Eqv7dL/9HQOM+bDMhxdEhlnZvPEDJrhNcWLKLmF8OaOoi1cbtdrslLCysUYYUSpggrZ21biBkglt9jU5trAaiI+C/Xu0xoLbf2IeBtxu0RKZFiOxzDZHzr2nqYtRKRD53uVxtoqOja9/Moh6Fta7fZTVM8+JyuYiIiKjzdZrdS2oReRDoB/yuhvTxIrJDRHZULPBlTDArKyubWVBQEHn+/PlWbrfbnvmYq6aqXLhwgcLCQt8SIXXRWD2IQsB/1tJ13nOViMj3gGnAUFW9WDUdQFUXAYvAM1Gu/otqTOPq27fv+o8//nhiXl7e06raAQgrKiqyiWzmqkRERJCcnExcXFydr9VYDcR2oIeIXI+nYRgNPOCfQUT6AAuBEap6svoljAldffv2XQ+sr/jeVgkwzUGjPGJS1TJgIp5fgFzg/1Q1W0T+U0RGebP9DogBVonIbhGptuG7McaYxtNog6BV9S3grSrnfuN3/L3GKosx9eky5vg8AjwGlAPFwPgqQ7yNaZaa3UtqY4KJ3xyfO4BewBgRqbrK3gpV/Y6q/iue+T2/b+RiGnNVrIEwpm58c3xUtQSomOPjo6pn/b6NxrM8tjHNns2zN6ZuLmuOj4g8BjwBRAK3Nk7RjKmboG4gdu7ceUpEalqvqT1wqoa0UGJxNpzO9XUhVV0ALBCRB4D/ADKq5hGR8cB477fFIlJEy/h/W6Gl1OUKTRnvZdXtoG4gVDWxpjQR2dEYm700NYuzyV3WHB8/rwIvB0rwn+MDzTrmBmHxNj/2DsKYuvHN8RGRSDxzfCoN0RaRHn7ffh841IjlM+aqBXUPwpimpqplIlIxx8cBLKmY44Nn39+1wETvKgGlwGkCPF4ypjkK5QZi0aWzhASLs4ldxhyfX17lpZttzA3E4m1mpL6WhTXGGBNa7B2EMcaYgEKygRCRESJyQEQOi8iTTV2euhCRAhHZ612faof3XIKIbBCRQ96v8d7zIiIveuP+RET6Nm3payYiS0TkpIjs8zt3xXGJSIY3/yERCYln+6FUf/2Fal32F3L1WlVD6oPnRWEe0BXPpKQ9QK+mLlcd4ikA2lc59zzwpPf4SeA57/FIPBstCTAQ2NrU5a8lriFAX2Df1cYFJABHvF/jvcfxTR1bHX8uIVV/q8QWknW5SjwhVa9DsQdxyaUPQsBdwB+9x38Efuh3Pks9PgLaikiz3E5NVTcDX1U5faVxDQc2qOpXqnoa2ACMaPjSN6iWUH/9BX1d9hdq9ToUG4hASx90bKKy1AcF3hGRnd6ZtgDJqnrCe/w5kOw9DvbYrzSuYI83kFCMqUJLqsv+grZeh/Iw11AxWFULRSQJ2CAi+/0TVVVFJOSGooVqXC1ci6zL/oItxlDsQVzp0gfNmqoWer+eBP6K5xHEFxXdbe/Xih34gj32K40r2OMNJBRjAlpcXfYXtPU6FBuISy59ECxEJFpEYiuOgduBfXjiqRjZkAGs8R6vBcZ5R0cMBM74dW2DwZXGtR64XUTivSNDbsdv284gFTL1118LrMv+grdeN/Vb/4b44BkdcBDPaJBpTV2eOsTRFc8olj1AdkUsQDvgXTxr+mwEErznBc/mNXnAXqBfU8dQS2x/Bk7gWX7iGPDw1cQF/BQ47P081NRx1dPPJiTqb5WYQrYuV4kzpOq1zaQ2xhgTUCg+YjLGGFMPrIEwxhgTkDUQxhhjArIGwhhjTEDWQBhjjAnIGghjjDEBWQMRgkRkhoj8b1OXw5j6ZnW7cVkDYYwxJiBrIIKciEwRkUIROefdZOb7wFTgfhEpFpE93nxtROR/ROSEN/8sEXF4034iIh+IyH+LyBkR2S8itzVlXMZY3W56tpprEBORFGAicJOqHheRLng2nHkG6K6qD/plX4ZnkbDuQDTwJp4lhRd60wcArwHtgXuA1SJyvapWXdvemAZndbt5sB5EcCsHnEAvEYlQ1QJVzauaSUSS8azvM1lVz6tnNc0/4FkIrsJJYK6qlqrqSuAA8P2GD8GYgKxuNwPWgwhiqnpYRCYDM4BUEVkPPBEga2cgAjghIhXnwqi8KUmhVl6Y6yhwbb0X2pjLYHW7ebAeRJBT1RWqOhjPL4oCz3m/+vsMuIhnP+C23k+cqqb65ekofr9hwLeA4w1ZdmNqY3W76VkDEcREJEVEbhURJ/A14ALcwBdAFxEJA1DPGvPvAP8lInEiEiYi3URkqN/lkoBJIhIhIj8Gvg281agBGeNldbt5sAYiuDmBZ4FTePa6TQKeAlZ504tE5GPv8TggEsgBTuN5aee/CfxWoIf3WrOBe1W1qKEDMKYGVrebAdsPwiAiPwF+5u3OGxMyrG7XjfUgjDHGBGQNhDHGmIDsEZMxxpiArAdhjDEmIGsgjDHGBGQNhDHGmICsgTDGGBOQNRDGGGMCsgbCGGNMQP8PfdFEvbnyQNsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# 绘制训练集和验证集的损失变化以及验证集上的准确率变化曲线\n",
    "def plot_training_loss_acc(runner, fig_name, \n",
    "    fig_size=(16, 6), \n",
    "    sample_step=20, \n",
    "    loss_legend_loc=\"upper right\", \n",
    "    acc_legend_loc=\"lower right\",\n",
    "    train_color=\"#e4007f\",\n",
    "    dev_color='#f19ec2',\n",
    "    fontsize='large',\n",
    "    train_linestyle=\"-\",\n",
    "    dev_linestyle='--'):\n",
    "\n",
    "    plt.figure(figsize=fig_size)\n",
    "\n",
    "    plt.subplot(1,2,1)\n",
    "    train_items = runner.train_step_losses[::sample_step]\n",
    "    train_steps=[x[0] for x in train_items]\n",
    "    train_losses = [x[1] for x in train_items]\n",
    "\n",
    "    plt.plot(train_steps, train_losses, color=train_color, linestyle=train_linestyle, label=\"Train loss\")\n",
    "    if len(runner.dev_losses)>0:\n",
    "        dev_steps=[x[0] for x in runner.dev_losses]\n",
    "        dev_losses = [x[1] for x in runner.dev_losses]\n",
    "        plt.plot(dev_steps, dev_losses, color=dev_color, linestyle=dev_linestyle, label=\"Dev loss\")\n",
    "    # 绘制坐标轴和图例\n",
    "    plt.ylabel(\"loss\", fontsize=fontsize)\n",
    "    plt.xlabel(\"step\", fontsize=fontsize)\n",
    "    plt.legend(loc=loss_legend_loc, fontsize='x-large')\n",
    "\n",
    "    # 绘制评价准确率变化曲线\n",
    "    if len(runner.dev_scores)>0:\n",
    "        plt.subplot(1,2,2)\n",
    "        plt.plot(dev_steps, runner.dev_scores,\n",
    "            color=dev_color, linestyle=dev_linestyle, label=\"Dev accuracy\")\n",
    "    \n",
    "        # 绘制坐标轴和图例\n",
    "        plt.ylabel(\"score\", fontsize=fontsize)\n",
    "        plt.xlabel(\"step\", fontsize=fontsize)\n",
    "        plt.legend(loc=acc_legend_loc, fontsize='x-large')\n",
    "\n",
    "    plt.savefig(fig_name)\n",
    "    plt.show()\n",
    "\n",
    "plot_training_loss_acc(runner, 'fw-loss.pdf')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "从输出结果可以看出准确率随着迭代次数增加逐渐上升，损失函数下降。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.5.6 模型评价\n",
    "\n",
    "使用测试数据对在训练过程中保存的最佳模型进行评价，观察模型在测试集上的准确率以及Loss情况。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 加载最优模型\n",
    "runner.load_model('best_model.pdparams')\n",
    "# 模型评价\n",
    "score, loss = runner.evaluate(test_loader)\n",
    "print(\"[Test] accuracy/loss: {:.4f}/{:.4f}\".format(score, loss))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 4.5.7 模型预测\n",
    "\n",
    "同样地，也可以使用保存好的模型，对测试集中的某一个数据进行模型预测，观察模型效果。代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 获取测试集中第一条数据\n",
    "X, label = next(test_loader())\n",
    "logits = runner.predict(X)\n",
    "\n",
    "pred_class = paddle.argmax(logits[0]).numpy()\n",
    "label = label[0][0].numpy()\n",
    "\n",
    "# 输出真实类别与预测类别\n",
    "print(\"The true category is {} and the predicted category is {}\".format(label, pred_class))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 4.6 小结\n",
    "\n",
    "本章介绍前馈神经网络的基本概念、网络结构及代码实现，利用前馈神经网络完成一个分类任务，并通过两个简单的实验，观察前馈神经网络的梯度消失问题和死亡ReLU问题，以及对应的优化策略。\n",
    "此外，还实践了基于前馈神经网络完成鸢尾花分类任务。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 4.7 实验拓展\n",
    "\n",
    "尝试基于MNIST手写数字识别数据集，设计合适的前馈神经网络进行实验，并取得95%以上的准确率。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
